What problem does this PR solve?

Problem Summary:

This PR implements a two-layer MS (Meta Service) RPC rate limiting system for Doris cloud mode BE:

1. Host-level rate limiting — Token-bucket based QPS limiter for all 21 MS RPC types, preventing a single BE from overwhelming the MS with burst traffic.
2. Table-level adaptive backpressure — When MS returns MS_BUSY error code, BE dynamically identifies and throttles the top-k highest-QPS tables using a state machine, and automatically relaxes limits after the pressure subsides.

Part 1: BE Host-Level Rate Limiting

Problem

In cloud mode, all BE nodes send RPCs (get_tablet, prepare_rowset, commit_rowset, etc.) to a shared Meta Service. A single BE experiencing load spikes (e.g., large batch imports, compaction storms) can send excessive RPC traffic that overwhelms MS, degrading service for all BEs.

Solution

Introduce HostLevelMSRpcRateLimiters — a per-BE, per-RPC-type rate limiter using token bucket algorithm.

Architecture:

• All 21 MS RPC types are enumerated in MetaServiceRPC enum (defined via X-macro for maintainability)
• Each RPC type has an independent TokenBucketRateLimiterHolder with its own QPS limit
• QPS limits are configured per CPU core: actual_qps = config_value × num_cores
• Thread-safe design using atomic_shared_ptr<RpcRateLimiter> array for lock-free concurrent access during limit() calls
• Each rate limiter includes a bvar::LatencyRecorder to monitor sleep durations caused by rate limiting

Configuration:

All QPS configs are mutable (DEFINE_mInt32), allowing runtime adjustment without restart. reset_all() re-reads configs and recreates rate limiters.

Integration:

Rate limiting is applied inside the retry_rpc() template function in cloud_meta_mgr.cpp, which wraps all MS RPC calls. The RpcRateLimitCtx struct carries the rate limiter reference. Rate limiting executes before each RPC attempt (including retries), with the call to apply_rate_limit() performing a bthread_usleep if the token bucket requires waiting.

New files:

• be/src/cloud/cloud_ms_rpc_rate_limiters.h / .cpp
• be/test/cloud/cloud_ms_rpc_rate_limiters_test.cpp

Part 2: BE Table-Level Adaptive Backpressure

Problem

Host-level rate limiting applies uniformly across all tables. When MS reports overload (MAX_QPS_LIMIT), it's often caused by a small number of high-traffic tables (e.g., tables with many concurrent stream load jobs). A uniform rate limit would unnecessarily penalize all tables, while the hot tables continue to dominate the RPC traffic.

Solution

Implement table-level adaptive throttling for load-related RPCs. When MS returns MAX_QPS_LIMIT, BE identifies the top-k highest-QPS tables and progressively reduces their QPS limits, while leaving other tables unaffected.

Scope: Only 5 load-related RPC types participate in table-level throttling:

• PREPARE_ROWSET
• COMMIT_ROWSET
• UPDATE_TMP_ROWSET
• UPDATE_PACKED_FILE_INFO
• UPDATE_DELETE_BITMAP

Architecture (4 components with clear separation of concerns):

MS_BUSY signal (MAX_QPS_LIMIT)
    │
    ▼
┌─────────────────────────┐       ┌──────────────────────────┐
│  RpcThrottleCoordinator │──────▶│  RpcThrottleStateMachine │
│  (timing control)       │       │  (pure state logic)      │
│  - upgrade cooldown     │       │  - upgrade history stack │
│  - downgrade trigger    │       │  - limit calculation     │
└─────────────────────────┘       └──────────┬───────────────┘
                                             │ Actions
                                             ▼
┌─────────────────────────┐       ┌──────────────────────────┐
│  TableRpcQpsRegistry    │       │  TableRpcThrottler       │
│  (QPS statistics)       │       │  (limit enforcement)     │
│  - per-table bvar       │       │  - StrictQpsLimiter      │
│  - top-k query          │       │  - per (rpc, table)      │
└─────────────────────────┘       └──────────────────────────┘

Component details:

1. TableRpcQpsRegistry — Tracks per-(rpc_type, table_id) QPS using bvar::PerSecond<bvar::Adder>. Supports efficient top-k query via min-heap. Configurable time window via ms_rpc_table_qps_window_sec (immutable, default 10s).

2. RpcThrottleStateMachine — Pure state machine with no time awareness or side effects. Maintains upgrade history as a stack for clean rollback.

   • on_upgrade(snapshot): For each top-k table in the QPS snapshot, calculates new_limit = current_qps × ratio (first time) or current_limit × ratio (already limited), with a floor of ms_rpc_table_qps_limit_floor. Returns SET_LIMIT actions.
   • on_downgrade(): Pops the most recent upgrade from history. If the table had a prior limit, restores it (SET_LIMIT). If no prior limit, removes it (REMOVE_LIMIT).

3. RpcThrottleCoordinator — Timing control layer using tick counts (1 tick = 1 ms).

   • report_ms_busy(): Returns true if enough ticks have passed since last upgrade (cooldown).
   • tick(n): Advances time by n ticks. Returns true if downgrade should trigger (no MS_BUSY for downgrade_after_ticks).

4. TableRpcThrottler — Enforces QPS limits using StrictQpsLimiter (strict fixed-interval, no burst allowed). Each (rpc_type, table_id) pair has its own limiter. Returns the time point when the request may execute; the caller sleeps until then.

5. MSBackpressureHandler — Orchestrator that wires all components together:

   • on_ms_busy(): Called when retry_rpc receives MAX_QPS_LIMIT. Consults coordinator for cooldown, builds QPS snapshot from registry, feeds to state machine, applies resulting actions to throttler.
   • before_rpc() / after_rpc(): Called around each load-related RPC for throttle enforcement and QPS recording.
   • Background tick thread: Runs every 1 second, advances coordinator by 1000 ticks. Triggers downgrade when enough time has passed without MS_BUSY.

Upgrade/Downgrade lifecycle example:

Time 0s:   MS returns MAX_QPS_LIMIT
           → Upgrade level 1: top-2 tables (A: 100 qps, B: 80 qps)
             → A limited to 50 qps, B limited to 40 qps

Time 2s:   MS returns MAX_QPS_LIMIT again (cooldown 5s not passed)
           → Skipped

Time 6s:   MS returns MAX_QPS_LIMIT (cooldown passed)
           → Upgrade level 2: top-2 tables now (A: 50 qps, C: 60 qps)
             → A limited to 25 qps, C limited to 30 qps

Time 11s:  No MS_BUSY for 5s
           → Downgrade: undo level 2
             → A restored to 50 qps, C limit removed

Time 16s:  No MS_BUSY for 5s
           → Downgrade: undo level 1
             → A limit removed, B limit removed

Configuration:

Observability (bvar metrics):

• ms_rpc_backpressure_upgrade_count / _60s — Upgrade event counts
• ms_rpc_backpressure_downgrade_count / _60s — Downgrade event counts
• ms_rpc_backpressure_ms_busy_count / _60s — MS_BUSY signal counts
• ms_rpc_backpressure_throttle_wait_<rpc_name> — Per-RPC-type throttle wait latency
• ms_rpc_backpressure_throttled_tables_<rpc_name> — Number of throttled tables per RPC type

New files:

• be/src/cloud/cloud_throttle_state_machine.h / .cpp
• be/src/cloud/cloud_ms_backpressure_handler.h / .cpp
• be/test/cloud/cloud_throttle_state_machine_test.cpp
• be/test/cloud/cloud_ms_backpressure_handler_test.cpp

Also renamed (not part of the feature, cleanup):

• common/cpp/s3_rate_limiter.h/.cpp → common/cpp/token_bucket_rate_limiter.h/.cpp (more general naming since it's now used beyond S3)

Part 3: System Table for Table-Level Throttler Observability

Problem

The table-level backpressure system operates transparently inside BE. When issues arise, users and DBAs have no way to inspect which tables are being throttled, what their QPS limits are, or what their current QPS is — beyond checking raw bvar metrics.

Solution

Add a new system table information_schema.backend_ms_rpc_table_throttlers that exposes the real-time state of the TableRpcThrottler on each BE. This table is a Backend-Partitioned Schema Table, meaning each BE reports its own throttling data, and queries are
distributed to all alive BEs and aggregated.

Schema:

Usage examples:

-- View all currently throttled tables across all BEs
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers;

-- View throttled tables on a specific BE
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers WHERE BE_ID = 10001;

-- Find the most severely throttled tables
SELECT * FROM information_schema.backend_ms_rpc_table_throttlers ORDER BY QPS_LIMIT ASC;

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.
    • Added host-level token-bucket rate limiting for all MS RPCs (enabled by default via enable_ms_rpc_host_level_rate_limit)
    • Added table-level adaptive backpressure handling triggered by MS MAX_QPS_LIMIT response (disabled by default via enable_ms_backpressure_handling)

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label